Cross-Layer Distillation with Semantic Calibration

Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training student to obtain better generalization ability. Existing studies mainly focus particular representation forms between manually specified pairs teacher-student layers. However, semantics may vary in different networks and manual association might lead negative regularization caused by semantic mismatch certain layer pairs. To address this problem, we propose Semantic Calibration Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target the each with an attention mechanism. With learned distribution, distills contained multiple rather than single fixed from appropriate cross-layer supervision training. Consistent improvements over state-of-the-art are observed extensive experiments various network architectures models, demonstrating effectiveness flexibility soft mechanism distillation.